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Abstract 

A theory-based instrument for measuring the quality of decisions made using any form of decision technology, including 
both decision-aided and unaided clinical consultations is required to enable person- and patient-centred care and to 
respond positively to individual heterogeneity in the value aspects of decision making. Current instruments using the 
term 'decision quality' have adopted a decision- and thus condition-specific approach. We argue that patient-centred care 
requires decision quality to be regarded as both preference-sensitive across multiple relevant criteria and generic across all 
conditions and decisions. MyDecisionQuality is grounded in prescriptive multi criteria decision analysis and employs a 
simple expected value algorithm to calculate a score for the quality of a decision that combines, in the clinical case, the 
patient's individual preferences for eight quality criteria (expressed as importance weights) and their ratings of the 
decision just taken on each of these criteria (expressed as performance rates). It thus provides an index of decision 
quality that encompasses both these aspects. It also provides patients with help in prioritizing quality criteria for future 
decision making by calculating, for each criterion, the Incremental Value of Perfect Rating, that is, the increase in their 
decision quality score that would result if their performance rating on the criterion had been 100%, weightings 
unchanged. MyDecisionQuality, which is a web-based generic and preference-sensitive instrument, can constitute a 
key patient-reported measure of the quality of the decision-making process. It can provide the basis for future decision 
improvement, especially when the clinician (or other stakeholders) completes the equivalent instrument and the extent 
and nature of concordance and discordance can be established. Apart from its role in decision preparation and evalu- 
ation, it can also provide real time and relevant documentation for the patient's record. 
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Background 

The increase in the range of options available for health 
and disease management, coupled with the shift 
towards greater patient involvement in recent years, 
has led to a profusion of decision aids and related sup- 
port systems aimed at patient, chnician and medical 
team.'""' This has been foUowed by the development 
of instruments and checkhsts to assess the quahty of 
such aids. Most have focused on the internal quality 
of the particular decision aid as appraised by a set of 
normative criteria, with the International Patient 
Decision Aid Standards instrument (IPDASi)'' emer- 
ging as the most prominent of such checkhsts. Where 
empirical evaluation has been undertaken or is pro- 
posed, most attention has been on particular process 
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and outcome aspects (e.g. acceptability, involvement, 
conflict, knowledge), rather than on the comparative 
performance of aided and unaided decision making in 
relation to overall decision quality. (A list of evaluation 
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measures is available on the Ottawa Patient Decision 
Aids website, http://decisionaid.ohri.ca/eval.html) 

None of the chnical trials included in the Cochrane 
systematic review of decision aids has evaluated the 
quahty of the decisions derived from different decision 
technologies (including decision aids) using a single 
index measure of overall decision quahty. There is 
therefore a need for evaluation measures that address 
the overaU quality of decisions, as distinct from meas- 
ures that address particular aspects of decision making. 

Adopting the Berwick^ philosophy of patient-centred 
care implies that all decisions should be regarded as pref- 
erence-sensitive, with the relative importance a patient 
attaches to various outcomes and processes having a 
large influence on, if not determining, what is decided. 
This philosophy also necessitates that decision quality be 
regarded as preference-sensitive and that the 
relevant preferences are those of the patient facing 
the decision, as opposed to the average preferences 
of a group of patients with the same condition or 
those of the health professional(s) involved in the 
decision. 

In the case of drugs or medical devices the main 
purpose of the intervention is to achieve a health bene- 
fit. It follows that the primary outcome of intervention 
research is to determine its effectiveness in accomphsh- 
ing that task. We suggest that, in the case of decision 
aids, the main purpose should be to enhance the overaU 
quality ('goodness') of the aided decision relative to 
that which can be achieved without it - by some 
other 'decision technology' - and that this should be 
assessed at the time of decision making and not in terms 
of any subsequent change or outcome.* Hence, decision 
quality should become the primary outcome of studies 
comparing decision-aided and unaided practices. This 
essay therefore argues for the use of decision quahty as 
a directly measurable patient-reported measure for all 
health conditions. It could be regarded as a Patient- 
reported outcome measure if the decision is regarded 
as an outcome of the decision-making process^ or, 
alternatively and more conventionally, as a patient- 
reported experience measure.^ While pohcy makers 
and researchers may benefit from a generic measure, 
the fundamental reason for it is so that the individual 
is able to assess the quality of the various health-care 
decisions they face in, and through, their hfe, using the 
same instrument, whatever the context, condition, 
timing in life or role occupied. We describe and illus- 
trate the application of the MyDecisionQuality (MDQ) 
instrument to this task. 

The development of MDQ fohowed an assessment 
of the available instruments for evaluating decision aids 
- mainly on the Ottawa website - which was designed 
to establish whether any of these instruments generated 
a generic and preference-based index of overall decision 



quality; as opposed to ones that were: (i) condition-, 
setting- or decision-specific; or (ii) measured one or 
more possible aspects of decision making such as pre- 
ferred involvement in decision,' satisfaction with the 
decision'" or decision conflict experienced," rather 
than overall decision quality or (in) did not weight 
their components to produce an index measure 
(i.e. were profile instruments) or, if they did enable 
weighting, did not elicit weights from the specific 
patient on the specific decision occasion. 

None of the instruments identified in the search con- 
stituted such a personalized preference-based measure 
of decision quality. The only instruments uncovered 
that used the label 'decision quahty' per se were those 
developed by Sepucha and colleagues.'^ Their condi- 
tion- and decision-specific decision quahty instruments 
(DQIs) include items that assess: (i) knowledge - the 
extent to which the patient was 'weh informed', (ii) 
concordance - the level of agreement between the 
patient's goals and concerns and their treatment and 
(hi) involvement - the extent to which the patient was 
involved in decisions about their care. In addition to 
the fact that they are not generic, these DQIs are not 
preference-based. The scores that are produced relate to 
particular segments of the DQI and are not aggregated, 
by weighting, into a single overall index measure of 
decision quahty for the individual patient. Moreover, 
the score for the concordance component is calculated 
only for patient populations, not for any specific 
patient. 

We do not regard the Decisional Conflict Scale" and 
its 4-item form SURE'"* as measures possessing con- 
struct validity in relation to decision quahty because 
they penahze decision processes and support systems 
that leave the decision maker in a state of warranted 
equipoise. 

Decision satisfaction is not an appropriate measure 
for assessing the quality of a decision aiding or making 
process, and will often reward suppression of uncertain- 
ties rather than their expression.''' For example, item 
two of the widely used Satisfaction with Decision 
instrument ('The decision I made was the best possible 
choice for me') makes it unsuitable in the evaluation of 
decision quality because it denies the possibihty that 
there may be two or more best possible choices.'" We 
regard patient empowerment as incompatible with the 
notion that the role of the physician is to provide a 
confident and single recommendation. 

Defining decision quality 

We take the view that both the definition and measure- 
ment of decision quality should be treated as preference- 
sensitive. Accordingly, in principle the measurement of 
decision quahty will require the decision owner to 
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choose which criteria to include in the instrument. 
A DQI based on a set of consensus-led items and 
weights (including the equal weights imphed in most 
checkhsts) seems incompatible with truly personalized 
patient-centred care. While Edwards and Elwyn'^ raise 
a number of legitimate issues concerning the operatio- 
nalization of decision quality, many of these stem from 
their view that a 'good decision' exists, but is yet to be 
defined. In contrast, we take the view that decision qual- 
ity - defined tautologically as the goodness of a decision 
- does not exist and should not be defined in a positiv- 
istic way. 'Decision quahty' is a multicriterial construct 
and all we can do, given the necessity to assess it, is to 
propose a set of items that appeal to our - and others' - 
va/Me judgements as to what should be included. (In this 
respect it parallels constructs like 'health-related quality 
of hfe' that instruments such as EQ-5D simultaneously 
define and measure.) Beyond this immediate challenge, 
the next task that any one constructing a DQI faces is 
operationalizing the measurement and synthesizing its 
components. We interpret Edwards and Elwyn as agree- 
ing with us that abdicating from this task because of the 
substantial operational challenges it poses is not an 
option, and we regard 'subjective' numerical 
calculation as a vital complement to their 'subjective' 
verbal dehberation approach in responding to these 
challenges. 



Developing a personalized and generic 
decision quality measure 

MDQ is a web-based generic instrument for the indivi- 
dual-specific measurement of decision quality based on 
Multi-Criteria Decision Analysis (MCDA). 

Dolan recently explored the potential of MCDA as 
the basis for decision support systems in health care, 
including Shared Decision Making (SDM).'^ '^ He out- 
lined the current portfolio of multicriteria methods and 
commented on their respective merits and problems. His 
assessment led him to favour the Analytic Hierarchy 
Process (AHP) because it offered greater flexibility, 
ease of use and strength of measurement compared 
with the other methods. However, he acknowledged 
that one of the main problems with using the AHP in 
chnical decision support is that its pairwise comparison 
elicitation processes, the main source of its strength in 
measurement, is both time consuming and cognitively 
demanding. 

In conjunction with the online survey program called 
Ehcia© in which it is embedded, Annahsa© forms an 
interactive, online decision support template that was 
explicitly designed to make MCDA-based decision sup- 
port less temporally and cognitively demanding.'^ 
It was not developed solely with applications in the 



health-care setting in mind, but it was designed to be 
the basis for tools which were practical in both time and 
resource-pressured situations - as weU as relaxed ones. 
Annalisa meets one of Bates et al.'s'^ commandments 
for the satisfactory delivery of a decision aid in stressing 
visualization and presenting all aspects of the decision 
(preferences, evidence and options) on a single screen. 
Annalisa-based decision aids provide a structured ana- 
lytical framework for decision dehberation and hence 
for the balancing of the two main contributors to deci- 
sion making - analytical modelling and intuitive judge- 
ment. ^° Above all, Annalisa tools seek the requisite 
balancing of normative rigour and operational practi- 
cahty.^''"" A study involving an Annalisa tool for pros- 
tate cancer screening has confirmed its ease of use.""* 

MDQ is a dually personalized DQI based on MCDA 
and currently implemented in Annahsa©, though in 
principle it could be implemented in any form of 
onhne spreadsheet. (By saying MDQ is based on 
MCDA we mean that there is always an inipHcit alter- 
native decision process (option) with which the MDQ 
result for the currently implemented option should be 
compared.) The assessor (e.g. patient) is responsible not 
only for (i) weighting the criteria of decision quaUty in 
terms of their relative importance but also (ii) rating the 
quality of a decision just made on the criteria. MDQ 
is generic in the sense that the criteria are phrased with- 
out reference to any particular decision or context. 
Information relating to the specific decision (such as 
one in a particular health-care setting and population) 
is to be provided outside the MDQ instrument, but in 
the larger decision support system in which MDQ will 
often be situated.'^ 

As with all implementations of the simple 'weighted- 
sum' version of MCDA, MDQ combines a set of 
importance weights for multiple criteria with perform- 
ance ratings for each option on these criteria and cal- 
culates the overall score as the expected value of these 
components. The patient's weightings for the eight cri- 
teria of decision quality are elicited as early as possible 
in the decision-making process and their ratings on how 
well the decision made performed on these criteria as 
soon as possible after it was made. The MDQ Score, 
unique to the patient and to the particular occasion, is 
automatically calculated as the summed multiplication 
of criterion weightings and ratings. A worked example 
is provided in Figure 1. 

Both MDQ and the decision aids developed in 
Annalisa are accessible via the internet from any oper- 
ating system, browser and on mobile tablets (iPad, 
Android). The resulting summary picture of the deci- 
sion quality assessment (showing Weightings, Ratings 
and MDQ Score) can be printed and/or downloaded as 
an image for later use, including sharing and formal 
clinical documentation. 
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Figure I . Screen capture of Annalisa presentation of MDQ for patient PCS2880. 
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Figure 2. Screen capture of MyDeclsionQuality weightings items (Ratings Items are rephrased in past tense). Scale: 0=Not at all 
important, 5 = moderately important and IO = extremely important. 



The desire to make MDQ practical in pressured situ- 
ations such as a heahh-care chnic determined the 
number of criteria included. The number that an indi- 
vidual could realistically be asked to weight and rate at 
the time of decision making was initially set at 10. The 
review of the most commonly used instruments in rela- 
tion to patient involvement and participation in health 
decision making helped us to generate a list of candi- 
date criteria. This list was reduced to 10 on the basis of 
either conceptual redundancy or inappropriateness for 
inclusion in a universal (i.e. not specifically health) deci- 
sion quahty measure. These 10 included six items which 
remained when it was later decided that eight was the 
maximum practical number for a user to weight and 
rate at the time of decision making and hence the max- 
imum number of items to include in a decision quality 
measure. This number is within Miller's magical 
number seven plus or minus two and is endorsed in 
the Cochrane Handbook.^"* The shorthand labels for 
these six criteria are: 'Options', 'Effects', Importance', 
'Trust', 'Control' and 'Commitment'. Of the remaining 
four items in the original 10, an Uncertainty criterion 
was subsumed in a 'Chances' criterion and an 



Emotional Support item in a general 'Support' 
criterion. 

Of the eight criteria in the current version of MDQ 
(Figure 2), the first four match the structural require- 
ments for any MCDA implementation in any context 
(Options, Criteria, Weightings and Ratings). These cri- 
teria also appear, in one form or another, in all check- 
lists for developing decision aids for health decisions, 
including IPDASi. The last four criteria relate to other 
aspects of the decision process and are also explicitly or 
implicitly included in most checkHsts for decision aids. 
The 'Commitment' criterion creates for investigating 
concordance at the point of decision and correspond- 
ence with future actions and outcomes. 

As with all the existing instruments referred to ear- 
lier there is no intention in MDQ to capture or assess 
the subjective experience of the patient (fear, anxiety, 
etc.). The patient expresses their views as to the support 
they received in relation to their feelings and emotions - 
and all other aspects of the decision experience - by 
their weighting and rating on the Support criterion. 

After discussions of the provisional eight items with 
immediate colleagues, we uploaded an online survey 
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Figure 3. Screen capture of Annalisa presentation of MDQ for patient PCS2880 showing MDQ Score breakdown and highlighting of 
part-worth, weighting and rating for Options. 



incorporating tlie initial MDQ on the Facebook page 
of tlie SDM Group, and emailed an invitation to com- 
ment to those on the lists of the Society for 
Medical Decision Making (SMDM) and Society of 
Judgment and Decision Making in mid-December 
2011. Allowing for crossover, we estimated that this 
provided us with a few hundred potential respondents. 
Twenty individuals completed the questionnaire (latest 
in mid-January 2011) and nine also provided comments 
on the MDQ screen. Their feedback was incorporated 
in the re-development of the MDQ, when it was com- 
patible with the underlying framework and construct. 

Figure 1 shows the Weightings component of MDQ 
as now adopted. The equivalent Ratings component of 
MDQ is not presented here, but it is identical to the 
Weightings except that the descriptions are phrased in 
the past tense (e.g. T was clear about the Options avail- 
able to me and the processes they involve'). 

While it is possible to have Users enter their weight- 
ings and ratings directly into Annalisa, early testing of 
MDQ using a convenience sample of academics showed 
that it was easier for their data to be elicited on a 0-10 
scale in the survey program (Elicia) within which the 
AnnaHsa for MDQ is embedded. The responses were 
then mapped directly on to a 0-1 scale and ported into 
Annahsa using the software bridge between the two 
programs. 

There are always tensions between what can be 
expected from development of a practice-relevant tool 
and one that is also used for research. Since the first 
(Weightings) part of MDQ is designed to be adminis- 
tered as early as possible in the decision-making process 
it constitutes an intervention in itself, whether or not 
any other intervention (e.g. decision aid) is involved 
before the Rating part is administered. It is not at all 
clear what should be the primary outcome, in terms of 
decision quahty, of a trial of MDQ-supplemented deci- 
sion making and standard practice. The challenge of 
vahdating a patient-specific, preference-based instru- 
ment such as MDQ does not appear to have been 



addressed in the literature thus far and we continue 
to seek assistance in this respect. Given the 
personalized character of MDQ, we are particularly 
interested in exploring the use of N-of-1 study 
designs.^* 



Illustration of the application of MDQ 

Figure 1 presents the MDQ screen of a patient from a 
randomized controlled trial (RCT) using two Annahsa- 
based decision aids for prostate cancer screening in 
Australia. The Weightings shown are as they were 
entered in Elicia in non-normalized format, that is, 
they do not add to one. They are normalized in 
Annalisa to add to one as the Score calculation is 
always using normalized weights. 

After seeing the MDQ Score, the patient has the 
option of viewing its breakdown into the contributions 
made by each criterion, that is, their 'part-worths', to 
estabhsh whether, and if so how, they might be able to 
improve their MDQ Score in the future, provided the 
opportunity exists. In Figure 3, the MDQ Score for 
the patient is partitioned into eight segments. If a seg- 
ment is relatively large then the criterion is making a 
larger contribution to the individual's MDQ Score. 
On the other hand, if it is relatively small then it is 
making a smaller contribution to their MDQ Score. 
By placing the cursor over any segment of the MDQ 
score bar, the relevant criterion label and the weight- 
ings and ratings for it are highlighted (column 1 in 
Figure 3; video demonstration at http://bit.ly/ 
IVyKWNm) If the patient has assigned the criterion 
a small weighting but also a low rating there is no 
need for concern. However, if they have assigned the 
criterion a small rating but a moderate or large 
weighting, they may want to think about how they 
might change things for the better, for example, by 
prioritizing the seeking of more information about 
the effects, more value clarification about criterion 



Kaltoft et al. 



115 



Priorities for Decision Quality Improvement .. ^~M^ 



Scores 








OPTIONS 






0.2S4 










tFFtCTS 






a.iS2 


IMPORTANCE 






0.109 


CHANCES 






0.109 


TRUST 


■ 




0.0SI 


SUPPORT 






0.152 


CONTROL 






0.065 


COMMITMENT 






0.109 


@ Weightings 




MCREHtNTAl EXPECTED VALUE Of PtRTECT RATING 


> 


(o-' Ratings 






> 



Figure 4. Incremental (Expected) Value of Perfect Ratings for each criterion for respondent PCS2880, shown as priority distribution. 
The longest bar indicates biggest gain from perfect rating. 



importance, moving towards their preferred level of 
control or whatever the criterion relates to. 

This feedback can be formalized in the form of the 
Incremental (Expected) Value of Perfect Rating (IVPR) 
(see Figure 4). The IVPR for a criterion indicates the 
amount by which the overall MDQ Score could be 
increased if, given the respondent's weighting of a par- 
ticular criterion, they achieved a (perfect) rating of one. 
In the case of this patient, this would suggest priority 
being given to improving the Rating for Options, fol- 
lowed by Effects and Support. 



Discussion 

As we move towards patient-centred care it is import- 
ant that we respond positively and wholly to patient 
heterogeneity in the value aspects of decision 
making. Developing a portfoHo of instruments to 
evaluate the overall quality of decisions in a transparent 
and preference-sensitive manner is a growing area of 
research. MDQ is offered as one that may be able to 
contribute to this process. 

Given its preference-sensitive nature how might 
MDQ be used to advance patient-centred research? 
Patients are heterogeneous in both their biophysical pro- 
file and their preferences. Because of the preference-sen- 
sitive nature of the instrument, the usefulness of average 
results (Weightings, Ratings and MDQ Scores) from 
MDQ depends on the purpose of using it in the first 
place. If a patient uses MDQ to assess the quahty of a 



decision made after using an onhne decision aid and 
prints out the MDQ screen, this may be used during a 
clinical consultation foUow-up to form the basis of a 
tailored conversation about particular aspects of the 
decision. In this case, it is only the individual patient's 
Weightings, Ratings and Score that matter. On the other 
hand, for an RCT involving simple and complex ver- 
sions of a decision aid, average MDQ scores may be 
used to compare decision quahty across the two arms 
of the trial. However, these results still need to be care- 
fully interpreted in light of the personalized criterion 
weightings they embody. In principle, the average results 
matter most where health-care resources are being con- 
sumed or a relevant group- or population-level policy 
decision is to be taken. In this respect, cluster and 
latent class analysis^** may be used to estabhsh meaning- 
ful patterns of preferences in the community,^' leading 
to an 'epidemiology of preferences'. 

In relation to the Annalisa-based decision aids we 
are developing, we envisage an alternative route 
where the individual selects the criteria most important 
to them from a longer list rather than having a set of 
criteria provided by researchers. We refer to this as a 
'Pick Your Own' version of a decision aid. This version 
formed one of the arms of the clinical trial involving 
two decision aids for prostate cancer screening.'"' 
A 'Pick Your Own' version of (MDQ) captures the 
essence of person-centred care but it is essential 
that any such alternative patient-reported MDQ meas- 
ure retains its theoretical and prescriptive basis 
in MCDA. 
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